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Semiannual  Report  (April-September  1990) 

/  OPTICAL  NEURAL  NETS  FOR  SCENE  ANALYSIS 

1.2  APPROACH 

Our  approach  is  hybrid  and  multidisciplinary.  We  marry  pattern  recognition  and  neural 
net  techn;nues.  We  also  marry  optical  and  digital  technologies.  A  hybrid  neural  net  thus 
results.  We  also  concentrate  on  the  use  of  one  basic  hybrid  architecture  that  is  useful  for 
implementing  various  optimization  and  adaptive  neural  nets.  Our  work  thus  distinguishes 
between  these  two  general  classes  of  neural  nets  (optimization  and  adaptive)  with  both  being 
realizable  on  the  same  basic  hybrid  architecture.  •i?l  ......  r\^  6  ,  J _  J ,  r 

The  seven  neural  nets  we  have  considered  are  now  briefly  summarized. 

The  input  neurons  to  the  production  system  NN  are  facts  (antecedents  and  consequents). 
Objects  and  object  parts  are  used  in  our  initial  work.  Surface  types  for  object  parts  (cylinder, 
sphere,  valley,  ridge,  etc.)  can  also  be  used  in  future  work.  The  objects  are  typical  of  those 
present  in  various  scenes.  The  weights  define  the  rules.  These  are  initially  posed  as  if-then 
statements,  with  all  rules  written  as  the  AND  of  several  antecedents  and  the  OR  of  several  such 
sets  of  antecedents.  The  output  neurons  that  fire  represent  the  new  facts  that  are  now  learned  to 
be  true.  As  the  system  iterates,  it  learns  new  rules  and  infers  new  results  on  the  present  input 
data.  We  initially  consider  a  propositional  calculus  system  (with  all  parameters  being  exact 
terms)  and  then  plan  to  address  a  predicate  calculus  system  (with  parameters  being  variables) 
that  is  much  more  powerful. 

The  E  input  neurons  in  our  mixture  NN  each  correspond  to  the  fractional  amounts  of  E 
elements  present  in  a  mixture  of  elements  within  one  region  of  an  input  scene.  The  outputs  from 
two  matrix-vector  multiplications  are  combined  to  form  the  new  neuron  states.  After  a  number 
of  iterations,  the  final  neuron  states  denote  the  fractional  amount  of  each  element  present  in  the 
input  mixture. 

The  matrix  inversion  NN  produces  the  inverse  of  a  matrix  that  is  given  to  the  processor. 
To  calculate  the  inverse  X  of  a  matrix  Q,  we  realize  that  QX  =  I.  We  formulate  the  solution 
(the  elements  of  the  inverse  of  Q)  as  the  minimization  of  an  energy  function  .  We  solve  for  the  X 
that  minimizes  the  energy  function  on  a  neural  net.  The  matrix  elements  (weights)  in  this  NN 
have  an  attractive  block  Toeplitz  form  and  thus  acousto-optic  (AO)  architectures  should  be  very 
suitable  for  implementing  this  NN.  This  represents  the  first  AO  NN.  Since  matrix  inversions 
arc  required  in  many  pattern  recognition  linear  discriminant  function  designs  and  in  most 
adaptive  algorithms,  this  NN  should  have  general  computational  use  in  image  processing  (as  well 
as  in  adaptive  radar,  control,  etc.). 

The  cubic  energy  NN  for  MTT  takes  measurements  on  objects  in  each  of  three  frames  and 
it.  assigns  one  target  per  measurement  and  time  frame.  This  is  useful  for  time  sequential  scene 
analysis  to  associate  objects  (or  object  parts)  in  several  t  ime  frames. 


1.3  SUMMARY  OF  NEURAL  NETS  (NNs)  CONSIDERED 


The  quadratic  energy  MTT  NN  is  a  simplified  version  of  the  cubic  energy  NN.  It  processes 
pairs  of  time  frames.  The  resultant  optical  architecture  is  much  simpler  than  the  cubic  energy 
NN  and  significantly  reduces  component  requirements. 

The  symbolic  NN  combines  a  symbolic  correlator,  production  system  NN,  feature  extractor 
and  image  processing  NN.  Its  major  advantage  is  the  ability  to  process  multiple  objects  in  the 
field  of  view  (this  is  achieved  by  the  symbolic  correlator).  No  other  NN  has  this  ability.  It 
outputs  a  symbolic  description  of  each  region  of  the  input  that  denotes  which  generic  shapes  are 
present  and  their  location.  These  data  are  then  symbolically  encoded  and  fed  to  an  NN.  The 
NN  is  unique  because  of  its  symbolic  input  neuron  representation.  Alternatively,  the  locations  of 
regions  of  interest  in  the  input  scene  are  used  to  guide  the  positioning  of  window  functions  (for 
segmentation)  from  which  input  features  are  extracted  and  subsequently  fed  to  an  NN  for  object 
classification.  These  NNs  again  combine  pattern  recognition  and  NN  techniques. 

The  adaptive  clustering  NN  is  our  major  effort.  The  input  neurons  are  features,  the  hidden 
layer  neurons  are  prototypes  of  the  various  classes  of  objects  and  the  output  neurons  denote  the 
class  of  the  input  object.  Clustering  techniques  are  used  to  select  the  original  hidden  layer 
neurons  (we  allow  several  neurons  or  clusters  per  object  class)  and  hence  the  initial  input  to 
hidden  layer  weights.  These  represent  a  set  of  linear  discriminant  functions  (LDFs).  The  output 
neurons  define  the  class  of  the  input.  The  hidden  to  output  layer  weights  map  the  clusters  to 
classes.  Our  study  of  criterion  functions  determined  the  type  of  error  function  used  to  train  the 
NN.  Thus,  advanced  pattern  recognition  techniques  are  used  to  initialize  the  set  of  NN  weights. 
A  new  adaptive  NN  learning  algorithm  is  then  used  to  refine  and  improve  the  initial  weight 
estimates  and  to  produce  the  LDF  combinations  that  provide  the  nonlinear  piecewise 
discriminant  surfaces  finally  used.  This  is  the  adaptive  learning  stage.  This  new  NN  combines 
pattern  recognition  and  NN  techniques. 

1.4  YEAR  1  SUMMARY 

In  Year  1  (April  1989  -  March  1990),  we  advanced  six  new  optimization  neural  nets  (NNs). 
These  include:  a  mixture  NN  (with  an  imaging  spectrometer  case  study),  a  cubic  energy 
multitarget  tracking  (MTT)  NN,  a  quadratic  energy  MTT  NN,  a  symbolic  correlator  NN,  a 
production  system  NN,  and  a  matrix  inversion  NN.  We  also  advanced  a  new  adaptive  clustering 
neural  net  (ACNN). 

We  devised  a  hybrid  optical/digital  NN  hardware  architecture  and  began  fabrication  of  it. 
It  employs  a  digital  HNC  NN,  and  an  optical  NN  with  electronic  support. 

The  key  feature  of  this  NN  hardware  is  that  it  is  multifunctional  and  able  to  realize  all 
seven  of  our  optimization  and  adaptive  NNs. 
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2.  YEAR  2  EFFORT  (6  Month  Summary) 

2.1  YEAR  2  TASKS 

The  eight  Year  2  tasks  to  be  considered  follow.  These  are  briefly  discussed  in  Section  2.2. 

•  TASK  1:  Fabricate  initial  laboratory  hardware  system  (film  mask). 

•  TASK  2:  Demonstrate  initial  laboratory  hardware  system  for  selected  associative 
processor,  optimization  and  our  adaptive  NNs. 

•  TASK  3:  Perfect  the  new  error  diffusion  computer  generated  hologram  (CGH) 
interconnection  concept. 

•  TASK  4:  Address  capacity  and  real  time  extensions  of  the  laboratory  NN. 

•  TASK  5:  ACNN  distortion-invariant  feature  space. 

•  TASK  6:  ACNN  extensions. 

•  TASK  7:  New  matrix  inversion  NN  algorithm  studies. 

•  TASK  8:  New  predicate  calculus  NN  algorithm  studies. 

2.2  OVERVIEW 

Tasks  1,  3,  and  4  address  the  fabrication  of  our  laboratory  hardware.  Task  1  refers  to  the 
initial  system.  Task  3  addresses  the  interconnection  mask.  Task  4  involves  new  components. 

Task  2  considers  laboratory  demonstrations 

Tasks  7  and  8  represent  brief  studies  of  2  aspects  (specific  applications)  of  our  ANN. 

Tasks  5  and  6  consider  the  ACNN  and  are  our  major  emphasis. 

The  thrust  of  our  approach  has  been  to  assemble  a  multifunctional  ANN  (capable  of 
solving  a  variety  of  NN  problems).  We  have  achieved  this  by  simulation  and  have  produced 
several  laboratory  examples. 

2.3  FUTURE  PLANS 

This  present  approach  results  in  a  project  with  many  aspects  (many  different  ANNs)  and 
many  new  NN  algorithms.  To  provide  more  focus,  our  remaining  Year  2  effort  (and  all  of  Year 
3)  will  emphasize  laboratory  hardware  and  adaptive  learning  ANNs  (the  ACNN  and  variations 
of  it). 

2.4  PUBLICATIONS  (DOCUMENTATION) 

We  have  provided  a  plethora  of  papers  on  the  various  new  aspects  of  our  NN  work.  These 
are  noted  here,  summarized  in  subsequent  subsections  with  all  references  (papers)  sent  earlier. 

Our  multifunctional  ANN  [l]  (Section  2.5)  is  hybrid  optical/digital  hardware  and  provides 
a  system  that  solves  all  major  types  of  ANN  problems.  Other  optical  NNs  arc  not 
multifunctional 

The  hardware  for  our  system  [2]  (Section  2.0)  is  detailed  (circa  January  1000). 
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Most  ANNs  cannot  accommodate  multiple  objects  in  the  field  of  view.  The  Neocognitron 
cannot  easily  and  cost-effectively  achieve  this  [3].  Our  symbolic  correlator  ANN  (Section  2.7) 
can  achieve  this  and  is  the  most  efficient  type  of  ANN  for  this  vital  case.  Its  concept  [4], 
simulation  [5],  and  laboratory  verification  [6j  have  been  obtained.  This  task  has  been  terminated 
due  to  funding  constraints. 

Our  associative  processor  (AP)  ANN  work  resulted  in  a  new  H-K  AP  and  robust  versions 
of  it  [7],  These  aspects  are  presently  not  being  pursued  under  our  DARPA  contract,  due  to 
budget  and  research  direction  constraints  (see  Section  2.8). 

Our  optimization  ANN  work  has  resulted  in  new  formulations  of  these  NNs  as  a  matrix 
times  a  vector  plus  a  vector.  In  this  area  of  ANNs,  we  have  considered  image  spectrometry  [2] 
and  multitarget  tracking  ANNs,  plus  matrix-inversion  ANNs.  Our  matrix-inversion  ANN  study 
resulted  in  very  new  and  significant  algorithms  [8]  that  insure  that  an  NN  is  used  and  that  its 
number  of  computations  is  competitive  with  other  techniques  (Section  2.9). 

Our  ACNN  work  has  involved  4  papers.  These  involved  new  feature  spaces  [9,10]  (Section 
2.10)  and  our  ACNN  algorithm  [11,12]  (Section  2.11).  This  algorithm  removes  ad  hoc 
parameters  present  in  most  NNs  and  combines  pattern  recognition  and  neural  net  techniques 
with  a  vastly  more  efficient  number  of  iterations  resulting. 

Our  software  now  includes  the  calibration  and  control  for  our  hybrid  optical/digital  NN 
(Section  2.12)  and  all  input  neuron  representation  spaces  and  classifiers  plus  our  ACNN  (Section 
2.13). 


Our  new  predicate  calculus  NN  concepts  have  been  advanced  (Section  2.14)  and  placed  on 
hold  pending  laboratory  and  ACNN  results. 

Our  optical  laboratory  results  on  the  initial  laboratory  system  include  new  associative 
processors  (with  storage  greater  than  any  other  associative  processor)  and  multitarget  tracker 
optimization  ANN  laboratory  results  [7]  (Section  2.15). 

2.5  MULTIFUNCTIONAL  ANN  fl] 

Reference  jl]  details  our  architecture,  how  it  combines  optical  and  digital  NN  hardware, 
and  how  it  allows  solutions  of  all  major  NNs  on  one  hybrid  processor. 

2.6  HYBRID  OPTICAL/DIGITAL  HARDWARE  [2] 

Reference  [2]  details  our  hybrid  hardware.  It  consists  of  a  digital  NN  interfaced  to  an 
optical  NN  with  a  control  computer  providing  versatility. 

2.7  SYMBOLIC  CORRELATOR  ANN  [3,4,5,61 

This  most  unique  ANN  uses  an  optical  correlator  interfaced  to  a  production  system  4 
ANN  with  symbolic  encoded  ANN  inputs.  For  the  specific  case  we  considered,  the  multiple 
correlation  filters  were  formed  from  generic  object  parts.  We  provided  simulations  [5]  and 
laboratory  [6|  results.  We  also  showed  [3]  how  other  ANNs  (such  as  the  Neocognitron)  cannot 
easily  address  this  problem  of  processing  multiple  objects  in  the  field  of  view  in  parallel. 


2.8  ASSOCIATIVE  PROCESSOR  ANNs  [71 

VVe  have  developed  new  associative  processor  (AP)  ANN  algorithms  with  larger  storage 
capacity  than  any  other  AP.  We  have  also  demonstrated  their  optica!  realization  on  our  initial 
hardware  [7], 

2.9  OPTIMIZATION  ANNs  [7,8l 

We  have  formulated  new  optimization  ANN  algorithms  (matrix  times  vector  plus  vector) 
for  3  different  optimization  NNs.  We  have  demonstrated  the  multitarget  version  of  this  new  NN 
algorithm  [7],  analyzed  an  imaging  spectrometer  optimization  NN  [2],  and  developed  new  and 
practical  matrix-inversion  ANN  algorithms  [8], 

2.10  ACNN  FEATURE  SPACES  [9.101 

A  new  feature  space  NN  input  with  reduced  dimensionality  and  improved  invariance  and 
performance  has  been  devised  [9]  and  tested  [10). 

2.11  ACNN  [11,121 

Our  ACNN  has  been  detailed  and  demonstrated  [11,12],  It  performs  as  well  as  the 
backpropagation  ANN  with  no  ad  hoc  parameters  and  with  about  1000  times  fewer  iterations. 
It  is  characterized  by  the  hybrid  combination  of  pattern  recognition  and  neural  net  techniques 
with  new  linear  algebra  algorithms  used  to  provide  better  performance  and  convergence. 

2.12  SOFTWARE  SUPPORT  (June  1990  Report) 

The  software  support  for  our  hybrid  optical/digital  NN  has  been  produced  and 
documented. 

2.13  ALGORITHMS /TRAINING /FEATURE  SPACES  ON  DIGITAL 
NN  (October  1990  Report) 

We  have  encoded  all  major  feature  space  generation  and  our  ACNN  algori  .im  in  digital 
NN  hardware. 

2.14  PREDICATE  CALCULUS  ANN  (October  1990  Report) 

This  significant  advancement  to  our  original  propositional  calculus  ANN  has  been  detailed. 
The  steps  and  remaining  work  in  it  have  been  described  and  new  ideas  generated.  This  task  is 
presently  on  hold. 

2.15  OPTICAL  LABORATORY  RESULTS  [7] 

These  laboratory  results  on  our  initial  laboratory  system  included  our  new  associative 
processors  (with  larger  storage  than  any  other  associative  processor)  and  one  of  our  new 
optimization  NNs  (for  multitarget  tracking). 
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412-268-2464 

Dr.  Robert  D.  Burnham 

Technical  Director,  Integrated  Optoelectronics  Devices  Group 

AMOCO  Technology  Company 

AMOCO  Research  Center 

Warrenville  Road  and  Mill  Street 

P.O.  Box  3011,  F-4 

Naperville,  1L  60566 

Dear  Bob, 

Things  are  really  proceeding  well  here.  There  are  3  items  that  need  immediate 
attention. 

1.  I  don’t  have  a  new  1990-1991  consulting  agreement  effective  1  September 
1990. 

2.  Don  Holmgren  and  you  have  to  make  a  choice  on  the  LD 
focusing/collimation  specs  to  aim  for  and  the  LDs  to  be  used.  This  has  to 
be  done  pretty  soon  We  need  the  LD  angle  and  spacing  specs  (we  can’t  1-D 
collimate  the  present  10  stripe  array  as  the  LDs  are  too  close  together,  thus 
your  IQ  1991  goal  must  change)  especially  for  the  200  pm  wide  aperture  LD 
array,  and  the  entrance  angle  for  the  FO  and  crystal. 

3.  We  need  10K  to  cover  clean  room  etc.  overruns  we  discussed  and  need  a 
decision  on  (2)  to  proceed.  When  (2)  is  done,  I  can  send  you  a  new  P.O. 
from  my  company  for  these  CGHs. 


As  Always, 


David  Casasent 

George  Westinghouse  Professor 
Director,  CEODP 


cc:  E.  Schlesinger,  CMU 

DC:mjl 
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